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Abstract. We consider the (projective) representations of the group 
of holomorphic automorphisms of a symmetric tube domain V(BiQ that 
are obtained by analytic continuation of the holomorphic discrete series. 
For a representation corresponding to a discrete point in the Wallach set, 
we find the decomposition under restriction to the identity component 
of GL{Q). Using Riesz distributions, an explicit intertwining operator 
is constructed as an analytic continuation of an integral operator. The 
density for the Plancherel measure involves quotients of F-functions and 
the c-function for a symmetric cone of smaller rank. 

1. Introduction 

Let G be the identity component of the group of biholomorphisms of a 
irreducible bounded symmetric domain P. The scalar holomorphic discrete 
series of G can be realised in the space of holomorphic functions on this 
domain. By reproducing kernel techniques, M. Vergne and H. Rossi [VR76] 
have shown (see also |Ber751 IWal791 IFK94j ) that it has an analytic continu- 
ation as a family of (projective) irreducible unitary representations vTq of G, 
parametrised by the so-called Wallach set. Let r be the rank of the domain 
and d its characteristic number (cf. next section for a definition). Then the 
Wallach set is the union of the half-line a > (r — 1)^ and a discrete part 
consisting of r points / = 0, . . . , r — 1. When a > p — 1, where p is the 
genus of P, the representation spaces are weighted Bergman spaces. 

Let r be an antilinear involution of V. Then T> := is a totally ge- 
odesic submanifold, hence a Riemannian symmetric space, and G := GJ 
contains its group of displacements. Such a domain is called a real bounded 
symmetric domain. 

When one restricts an irreducible unitary representation of a group to a 
subgroup, the representation need not to be irreducible anymore, and the 
decomposition into irreducibles is called a branching law. In our context two 
branching problems have been extensively studied: the decomposition of the 
tensor product representation vrQ^vfo, and the restriction of tTq, to symmetric 
subgroups G = G^ where r is an antilinear involution oiT>. A formula for 
the first problem and for a > p — 1 was given without proof by Berezin for 
classical domains in [Ber78j . H. Upmeier and A. Unterberger extended it to 
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all domains and gave a Jordan theoretic proof [UU94] . The second problem 
was solved (for the same parameters) by G. Zhang and (independently) by 
G. van Dijk and M. Pevzner [ZhaOlat IvDPOlj . and also by Y. Neretin for 
classical groups |NerOOa| . Those two problems are in fact similar. The 
restriction map from V to V (resp. from V x V to P) gives rise to the 
Berezin transform on D (resp. V), which is a kernel operator. The solution 
then consists in computing the spectral symbol of the Berezin transform, 
or, if one prefers, in computing the Fourier transform of the Berezin kernel. 
In [ZhaOO] and in [vDPOlt Section 5] the problem of decomposing ■Ka'^Wa+i 
where i G N is also solved, by the same method. A similar problem is also 
studied in [FPOSj . 

For arbitrary parameters, those problems are more complicated, and no 
general method seems to apply. In j0Z97] the tensor product problem for 
G = SU (2, 2) is solved for any parameter. In [ZhaOlbj the representation 
7rd(8>7fd is decomposed for any G (vTd is called the minimal representation). 

2 2 2 

In |Ner02j . Y. Neretin solves the restriction problem from U{r,s) to 0{r,s) 
(r < s) for any parameter by analytic continuation of the result for large 
parameters. If r = s the support of the Plancherel formula remains the same 
for all a > r — 1 (here d = 2) but when s — r is sufficiently large new pieces 
appear when a crosses p — l = 2(r + s) — 1 and the situation gets worse as a 
approaches to (r — 1)|, as he had already explained in [NerOObj . For points 
in the discrete Wallach set, the situation is not clear. In his thesis the second 
author manages to decompose the restriction of 5*0(2, n) to SO{l, n) for any 
parameter [Sep07bJ , as well as the restriction of the minimal representation 
of SU{p, q) to SO{p, q) |Sep07a| , and the minimal representation of Sp{n, M) 
(resp. SU{n,n)) to GL+{n,R) (resp. to GL(n,C)) |Sep08| . 

Assume that T> is of tube type, i.e. that V is biholomorphic to the tube 
domain Tq over the symmetric cone Q. Then the inverse image of is a 
real bounded symmetric domain. In this paper, generalising |Sep08| , we 
establish, for any parameter in the discrete Wallach set, the branching rule 
for the restriction of the associated representation of G = Gq(Tq) to G = 
GLo{n). 

We use the model by Rossi and Vergne which realises the representation 
given by the l-th. point in the Wallach set as L'^{diil, fii), where diQ is the 
set of positive semidefinite elements in dfl of rank /, and /i/ is a relatively 
G-invariant measure on 9/17. A key observation is that for any x in 
the function g Ai^{g*x) on G, where Ajy is the power function of the 
Jordan algebra, transforms like a function in a certain parabolically induced 
representation. A naive approach to construct an intertwining operator 
from L'^^diil, fii) into a direct sum of parabolically induced representations 
would then to weight the functions above by compactly supported smooth 
functions, i.e., to consider mappings / i— > Jq ^ f{x)A^{g*x)dfj,i{x), for / 
in Cq°(5/0). It will become clear that this approach is in fact fruitful. 
However, there are two problems that have to be dealt with. First of all, 
it is not obvious that the natural target spaces are unitarisable. Secondly, 
and more importantly, the integrals above need not converge for the suitable 
choice of parameters v>. However, as we shall see, both these problems can 
be solved. 
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The paper is organised as follows. In Section 2 we recall some facts about 
Jordan algebras and symmetric cones that will be needed in the paper. In 
Section 3 we prove an identity between the restriction of a spherical function 
for the cone $7 to a cone of lower rank in its boundary and the corresponding 
spherical function for the lower rank cone. In Section 4 we define a class 
of irreducible unitary spherical representations that provides target spaces 
for the integral operators discussed above. These are constructed using the 
Levi decomposition of the group G by twisting parabolically induced unitary 
representations for the semisimple factor of G by a certain character. In 
Section 5 we construct the intertwining operator as an analytic continuation 
of the integral operator above. After this has been taken care of, a polar 
decomposition for the measure due to J. Arazy and H. Upmeier |AU03] 
allows to express the restriction of the intertwining operator to /C-invariant 
vectors in terms of the Fourier transform for a cone of rank /. Using this 
identification, the inversion formula for the Fourier transform can be used to 
prove the Plancherel theorem for the branching problem. In the appendix we 
provide a framework for certain restrictions of distributions to submanifolds 
which will be useful for giving an analytic continuation for the integral that 
should give an intertwining operator. It should be pointed out that the 
standard theory for restricting distributions (e.g. |Hor83l Cor. 8.2.7.]) does 
not apply to our situation since the condition on the wave front set for the 
distribution is not satisfied. Instead we have to use restrictions based on 
extending test functions in such a way that they are constant in certain 
directions from the submanifold (cf. Appendix . 

We finally want to mention that branching problems related to holomor- 
phic involutions of P have also been studied in [Rep 79 [ IKob98[ IBS021 IPZ04) . 

Acknowledgement. The authors would like to thank Karl-Hermann Neeb 
for enlightening discussions and comments that led to substantial improve- 
ment of the presentation. 

2. Jordan theoretic preliminaries 

Let y be a Euclidean Jordan algebra. It is a commutative real alge- 
bra with unit element e such that the multiplication operator L{x) satisfies 
[L{x) , L{x'^y\ = 0, and provided with a scalar product for which L{x) is 
symmetric. An element is invertible if its quadratic representation P{x) = 
2L(x)^ — L{x^) is so. Its cone of invertible squares O is a symmetric 
cone: it is homogeneous under the identity component, G, of the Lie group 
GL{Q) = {g € GL(y) \ gVt = fi}, and it is self-dual. It follows that the 
involution Q{g) := g^* := {g*)~^ (where g* is the adjoint oi g with respect 
to the scalar product of V) preserves G (which is hence reductive). The 
stabiliser K = Ge of e coincides with the identity component of the group 
Aut(y) of automorphisms of V and with the fixed points of G under the 
involution G, and hence is compact. Thus is a a Riemannian symmetric 
space. 

The tube Tq = V (Bi^ over 0, in the complexification of V is an Hermit- 
ian symmetric space of the non-compact type, diffeomorphic via the Cayley 
transform to a (tube type) bounded symmetric domain. Any element of 
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GL{0,), when extended complex-Unearly, preserves Tq. In this fashion G is 
seen as a subgroup of the identity component G of the group of biholomor- 
phisms of Tq. 

We assume that V is simple. Then there exists a positive integer r, 
called the rank of V, such that any family of mutually orthogonal minimal 
idempotents has r elements. Such a family is called a Jordan frame. Let 

be the Cartan decomposition of the Lie algebra q of G. Then the map 
X L{x) yields an isomorphism F — > p. The subspace generated by a 
Jordan frame (more precisely by the associated multiplication operators) 
is a maximal abelian subspace of p and conversely, any maximal abelian 
subspace of p determines (up to order) a Jordan frame. Prom now we fix a 
choice of a Jordan frame (ci, . . . , cv) and let 

and A = exp a. 

Any X in y can be written 

X = k ^ XjCj 

i<i<f 

where k ^ K and the Xj are real numbers, and the family (Ai,...,Ar) 
is unique up to permutation (its members are called the eigenvalues of x). 
Then x belongs to if and only if for all 1 < j < r, Xj > 0, and this spectral 
decomposition corresponds the KAK decomposition of G, the ^-component 
in the decomposition being unique up to conjugation by an element of the 
Weyl group W = &r G. The rank of x is defined to be the number of its 
nonzero eigenvalues. There exists on V a i^-invariant polynomial function 
A(x) (the determinant) and a ET-invariant linear function tr(a;) (the trace) 
that satisfy 

r r 

A(x) = n^^- and iv{x) = Y,Xj. 

The determinant defines the character 

(1) A(5) := A(3e) 

of the group G. 

A Jordan frame gives rise to the important Peirce decomposition. Since 

multiplications by orthogonal idempotents commute, the space V decom- 
poses into a direct sum of joint eigenspaces for the (symmetric) operators 
(L(cj))j=i....,r- The eigenvalues of L(c) when c is an idempotent, belong to 
{0, ^, 1}. Let us denote by V{c, a) the eigenspacc corresponding to the value 
a. The decomposition into joint eigenspaces is then given by 

y= Vij^ 

l<i<j<r 

where 

Vu = vici,i)n(~]v{cj,o), 
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We have Va = Mcj and the Vij all have the same dimension d, called the 
degree of the Jordan algebra. 

We can now describe the roots of (g, a). Let ((5j)j=i,...,r be the dual basis 
of {L{cj))j=\^,,,^'r in a*. Then the roots are 

_i_ , 6j ~ 5i 
afj = ±^y^' l<i<J<r, 

and the corresponding root spaces are 

Stj = \aeVij}, 

where xDy = L{L{x)y) + [L{x), L{y)]. Let N be the nilpotent subgroup 

N = exp 0+. 

l<i<j>r 

Then G has the Iwasawa decomposition G = NAK. 

For any idempotent c, the projection on 1^(1, c) is -P(c), and y(l,c) is a 
Jordan subalgebra, hence a Euclidean Jordan algebra with neutral element 
c (note that it is simple with rank the one of c). We denote by Oi(c) its 
symmetric cone. In particular for 

I 

k=l 

we set 

y(') = F(l,e;) and !^(') = Oi(eO, 
and also note G^'^ the identity component of K^'-^ = Gei and A^'^ 

the determinant of V^^\ The principal minors of V are then defined by the 
formula 

A(,.)(x) :=AW(P(e,)(.x)). 

Then x is in if and only if for all 1 < j < r, A(j)(x) > 0. Let G C and 
set for X in 0, 

AUx) = A^-'^^(x)A^ipnx) . . . a;';:;p^(x)a^;)(x). 

Using the basis (Sj), wc can identify i/ with an clement of a^. Then if a(g) 
is the projection of g on A in the Iwasawa decomposition, 

(2) A^{gx) = e'''°^^^a^A^{x). 

The action of G on the boundary dO, of 0, has r — 1 orbits, which may 
be parametrised by the rank of its elements. We denote by diQ, the orbit of 
rank I elements, i.e., 

diQ = Gei. 

There exists on diO, a unique relatively G-invariant measure ni, which trans- 
forms according to 

dfiligx) = A^{g)dni{x). 
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The Hilbert space associated to the Wahach point /| is, up to renormalisa- 
tion, isometric to L'^{diQ, fii) [FK94^ Theorem X.III.4], and the representa- 
tion of G in this picture is then given by 

(3) 7r\g)f = AT{g)f{g*.). 

The measures ni were constructed by M. Lassalle |Las87j and can also be 
obtain as Riesz distribution, thanks to S. Gindikin's theorem [FK941 VII. 3]. 
A major tool for our purpose will be the polar decomposition of fii |AU031 
Theorem 3.2.6]. Let 11/ = K.ei be the set of idempotents of rank /. Then 
difl is the disjoint union 

din= y n{u). 

UGH; 

Since elements of G permute the faces of 0, (which are of the form 0,{u) for 
idempotents u), an action is induced on 11;, such that the preceding equality 
defines a G-equivariant fibration 

For any function / in the space G^{diQ) of smooth functions with compact 
support on di^l, 

(4) [ fdfii= [ dk [ Aj{x)f{kx)d^S, 

where di x is the unique -invariant measure on 

The set diQ is not a submanifold of V. However let V>i be the (open) 
set of elements in V with rank bigger or equal than I. To any Z-element 
subset Ii C {1, . . . ,r} one can associate the idempotent e/, = Xlje/, '^j ^^'^ 
the minor A/;(x) := A(P(e7;)x + e — e/J. Then 

V>i= U {^e^l A/,(x)/0}, 

/,C{l,...,r} 

and [Las87l Propositions 3 and 7] show that diO, is a (closed) submanifold 
of V>i. 

3. An identity between spherical functions 
The spherical functions on 0, may be defined for v in by the formula 

$^(x) = / A,^{kx)dk. 
Jk 

When v satisfies Kz^i > • • • > > 0, a property that we will denote by 
3^1^ > 0, the generalised power function Ajy and the spherical function 
extend continuously to fi. Now let 

(5) ai = {L{cj),j = l,...,l) 

for 1 < / < r — 1 and assume that v belongs to (aic)* , i.e. that its (r — l) last 

coordinates vanish. Then u also defines a spherical function of Jl^'^ Let 
a be a real number. When v appears in the argument of an object related 
to V^''\ we will use the convention that u + a := {i^i + a, . . . ,1^1 + a). Recall 
that n^^^ cdiQ. 
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^Ux)=lii^^^\x), where ^(f) _ ^ ^(0 iT^^ n(0 ^ ^ 



Here T^^i) is the Gindikin Gamma function for the cone 

The theorem is proved in the case i^* G N' in [AU031 Proposition 1.3.2 and 
remark 1.3.4]. We use this result and the following lemma, which is based 
on Blaschke's theorem (see |KroOH Lemma A.l] for a detailed proof). 

Lemma 3.2. Let f be a holomorphic function defined on the right half- 
plane {z £ C \ > 0} . If f is bounded and f{n) = for n € N, then f is 
identically zero. 

Proof of the theorem. Let us set zj = Vj — i^j+i, j = 1, — 1, zi = vi, so 
that "Rzj > and uj = J2k=i ^k- Let x € fi^'^ and let 



'v>{zi,...,zi) 

Let us fix = m-j G N, j = 2, . . . , L If 6 > a > 0, one can see by Stirling's 
:h 

function 



formula that p^^^^^j is bounded on the right half plane. It follows that the 



r(z + EL2^A' + f) 

is bounded and hence also z ^ ^^^] ■,. Now 



(2,m2,...,mO (a;) I < / A|^2 (^^.^ _ ^ ^ (^^.^ 



K 



dk 



< sup ( {kx) . . . A™' {kx) ) ( sup A(i) {kx, 

and 



Ax] 



< sup f a;!!? (fcx) . . . A"?' {kx) \ I sup An ) [kx] 



Let b > sup^ A(i)(A;j;) > sup^(i) A(i)(A;x). Then the holomorphic function 
f{z) = F{z,m2, ■ ■ ■ ,'mi)6~^ is bounded and vanishes on N, hence on the 
right half plane, i.e., for every z G C with ^z > and ruj G N, 

F{z,m2, . . . ,mi) = 0. 

By the same argument one shows that for every zi G C with Kzi > and 
rrij G N, the map z i— > F(zi, z, ma, . . . , m;) vanishes identically, and the 
proof follows by induction. □ 
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4. A SERIES OF SPHERICAL UNITARY REPRESENTATIONS 

In this section we introduce a family of spherical unitary representations 
that will occur in the decomposition of L'^{di^) under the action of G. 
For 1 < / < r - 1 let 

i>i<j 

Note that it is a (nilpotent) Lie algebra and that (of. ([5])) 

ai®^L{e) = Pi keraj.. 
i<i<j 

The closed subgroup Zciai) = Zdai ©ML(e)) normalises Ni = expn^ hence 

Qi = ZaiaiWi 

is a subgroup of G. Moreover Zciai) HNi = {id} so this decomposition is a 
semidirect product. The Lie algebra of Qi, 

m = hi^i) e nz = m © a e flj © 0,^-, 

l<i<j l>i<j 

where m = 3t(o), is a parabolic subalgebra of g, and since Qi is the nor- 
maliser of q; in G [Kna02l 7.83], it is a closed subgroup of G (the parabolic 
subgroup associated to q;). It is also the stabiliser of a flag of idempotents 
(ei, 62, ... , ei). Note also that since V{L{cj), 1) = Mcj, we have 

(6) ZG{ai)cj =R+c,, j = l,...,l. 

Lemma 4.1. Let Ai = expa/ and 

Ml = dj^^ZG{ai)cy 
Then the multiplication map 

MixAixNi^ Qi 

is a diffeomorphism. 

Proof. It is clear from ([6|) that the product map M; x — > Zg{cii) is a 
smooth bijection hence it is a diffeomorphism. □ 

Note that the decomposition in the preceding lemma is not exactly the 
Langlands decomposition of Qi. However, it is more adapted to our pur- 
pose. We will let ai{q) denote the ^/-component of q £ Qi in the preceding 
decomposition. For v G (aic)*, let e^ 01 be the character of Qi defined 
by {Kgje" (g) l){q) = e'^i^s and let us denote by G{G,Qi, 1(g) (g) 1) 
the Frechet space of continuous complex valued functions on G that are 
Qrequivariant with respect to 1 ® e*^ iX) 1, i.e., 

G{G, Qu 1 ® e'^ 1) = {/ G C{G) \ Mq G Qu f{gq) = e-'^l°S"'('?)/(5)}. 

The induced representation IndiQ^il^e^ (gl) is the left regular representation 
of G on G{G,Qulg)e'' ®1). 

We will know determine values ol v £ (flic)* for which the representation 

Indg^(l(g)e''® 1) (g) A^T 

(cf. ([T|)) can be made unitary and irreducible. 
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The group G admits the Levi decomposition 

G = G' xR+ 

where the semisimple part G' is the kernel of the character A. Then K is 
a maximal compact subgroup of G' and the Lie algebra q' of G' has Cartan 
decomposition g' = 6 © p' with p' = {L{x) e p | tr x = 0}, and a' = a n p' is 
maximal abelian in p'. Let 



^<j<l l<i<j 

j=l+l ^ ^ l<i<j 



Then 



= © a, © n^. 



is a parabolic subalgebra of G' . The corresponding parabolic subgroup Q'l 
admits the Langlands decomposition 

Q\ = M[A\Ni, 

where A'^ = exp aj and (cf. |Kna02t Ch. VII, Propositions 7.25, 7.27 and 
7.82]) 

M/ = Zx(a;)exp(m;np'), 
whose Lie algebra is mj. For i/ G (oj^^)* the induced representation 

Indg',(l ©e'^(g) 1) 

is defined in the same way as for G. 

Lemma 4.2. (i) = x M+, 

(ii) Ml C Ml. 

Proof. To prove (i) we observe that since M+ C Qi, we can write Qi = 
Q' X M_|_, for some subgroup Q' C G' . Since Qi (resp. Q'l) is the normaliser 
of (resp. q^) in G (resp. G'), the inclusion Q'l C Q' is obvious and the 
converse follows from the fact that Ad(G) preserves g'. Since X.cj = for 
X G m'l and I < j < I, the assertion (ii) will follow from Zxiai) = Zxia'i). 
Let k £ Zxici'i), i.e., for j = 1, . . . , /, 

T (t( \ L{e-ei)\ L{e-kei) L{e - ei) 

(7) k. ^L{cj) - j = L{kcj) — ^ = L{cj) - . 

By summing over j one obtains 

L{kei) ^—;L{e - kei) = L{ei) ^—,L{e - e;), 

r — I r — I 

and hence L{kei) = L{ei). By ([?]) we then have L{kcj) = L{cj) for j = 
I,..., I, i.e., k G Zxiai). □ 

Let us denote by (5j)j=i.../ the dual basis of {L{cj) — )j=i...i in 

(a'l^)* . By 6j ^ 6j we define an isomorphism (a;c)* ~ (ajj^)*, v ^ V. 
Let TTijy be the character of M+ defined by 

(8) m^{C) = c'^'^^^^^'''. 
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Proposition 4.3. 

Indg^ (1 e*" ® 1) (g) A-f ~ Indg| (1 O 1) 

Proof. Let us denote by ai{q') the Aj-component of q' in the Langlands 
decomposition of QJ- Then for all q = q'^ & Qi = Q'l ^ 

^9) log a; (?) = gi' log (g') _ 

Indeed, if q' = m' e^'^^^^^^^^ ' is the Langlands decomposition of q' 
in Q'l^ then 

q ^ (j„'g(-7^+log?)i^(e-e!))gE^.ife)+log?L(ei)^ 

is the Langlands decomposition of q in Q;. If / G C(G, Q;, 1 e*^ 1), then 
by ([9]), its restriction / to G' belongs to the space C(G',QJ,1 ® e'^ ® 1). 
Conversely, if / € C{G', QJ, 1 (g) e*^ ig) 1), one obtains, again by Q, a function 
/ G C{G,Qu 1 e*" (g) 1) by setting /(^'C) := C'^'^'fig') and we obtain 
thereby a bijection C{G, Qz, 1 ® e*" (g 1) ~ C(G', Qj, 1 (g) e"^ (g) 1). Now the 
operator 

T : C{G, Qz, 1 g) e'' (g 1) (g C ^ C{G', Qj, 1 1) C, 

/ z 1-^ / g) z, 

intertwines the actions of G. Indeed, if / € C{G, Qi, 1 (g) e*^ (g 1) and g = 
g'C^G = G' X M+ then fljr^-) = f{g^-^-) = C^^^7(5'"^-)> and hence 

T ® A-f = /(?^) ® ^--i{g)z 

= J{g'''-)®Q-'^^^^^z. 

□ 

The representation IndQ^(l (g) e*^ (g) 1) (g) A~~ extends to a continuous 
representation (denoted by the same symbol) on the Hilbert completion of 
C(G, Qz, 1 (g e"^ (g 1) (g C with respect to 

|/(g,2|2 = ( f{k)dk\z\^ . 
Jk 

Lemma 4.4. For any m E M/ and any x , 

A(j)(mx) = A(j)(x), 1 < i < /. 

Proof. Let m E M^. Then for j = 1, . . . m commutes with L{ej), and 
m G Auty(-''), so 

A(j-)(mx) = A(^')(P(cj)(mx)) = A(^')(mP(cj)(x)) = A(J')(P(cj)(x)) 

= A(j-)(x). 

n 

Proposition 4.5. The map g A-^{g*e) is a (norm one) K -invariant 
vector in G{G,Qi, 1 (g) e*^ (g 1). 
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Proof. Let q & Qi and let q = man G MiAiNi. Then for g £ G, one has, 
since n* £ N and a* = a, 

A_^((gg)*e) = A_^{n*a*m*g*e) = e-'^^"^^") A_^(m*5*e) 

and because of Lemma 

A_,((ffg)*e) = e-'^'°s('^)A_,(<7*e). 

□ 

Let 

l>i<j j=l 
be the half sum of the negative O; © ML(e)-restricted roots (counted with 
multiplicities) . 

Theorem 4.6. For almost every A £ a* (with respect to Lebesgue measure), 

the representation IndQ^(l (8) e'' ^ 1) (8) A~~ with u = iX + pi + !j 
irreducible unitary spherical representation. 



Proof. First, let us remark that if u = iX + pi + ^ with A € then since 

rid , d „ /(/ + !) ,\ , 

E^^- - X = + 4 + ^ + " ^ 2^ " ^ 

the representation m^^ (cf. dS])) is unitary. We now claim that + ^ is the 
half sum of the negative oj-restricted roots (counted with multiplicities). 
Indeed, 

d / L{e-ei)\ 

i<i<j ^ / 

i<l<j \ k>l / 



d, Id Id f ^ , . Lie — eiY 

= -{r + l-2k) + - = p, + -[L{e,)-^^'^ 

The theorem now follows from Proposition l4.3l and Bruhat's theorem |vdB971 
Theorem 2.6]. □ 

Let us note p'l = Pi + ^-f ■ We now set 

(vr^,W^) :=Ind§^(lOe^'^+''J0l)0A-T, 

and 

Vt, := A_(i^+p/) ((•)*€) (g) 1. 

Let us compute the positive definite spherical function associated to vr^, 
that is, 

^-(ii/+p;)((5'~"^'''^*' 

K 



Hg) = Mg)vu,vu)u= / A_(,^+pM((5 ^k)*e)dkA i{g). 
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Since g~*e = Q{g)e = {ge)~^ |FK94t Theorem III. 5. 3], $ can be written, as 
a function on $7, 

<S>ix) = [ A_^i,^p,){x~')dkA-T(x) = $_(,,+p;)(x-i)A-T(x), 

J K 

and sin ce (x^^) = ^i,,+p'^+2p{x) with p = Yl'^j=i{2j - r - l)6j 

|FK94[ Theorem XIV.3.1 (iv)], 

d ^ 

^{x) = ^i^+r,i+p{x) where rji = - ^ {2j - I - r - l)6j. 

j=i+i 

Recall jl'K94l Theorem XIV.3.1 (iii)] that = if and only if 

I/' = wu for w G W . Hence the representations ttx and vry, with A, A' in a^, 
are equivalent if and only if zA' + = w{iX + ri{). Since is real it follows 
that TTy is equivalent to ttx if and only if A' = w\ with w ^Wi := Gi. 

5. The intertwining operator and the Plancherel formula 

Let G C' such that ^{-{iu + p[)) > 0. Then for / G C^idiQ) and 
g & G, the formula 



(10) T,/(5) = / /(x)A_(,,+v)(5*x)dw(x) 

defines a continuous function on G. Moreover, it follows from ([2]) and 
Lemma 14.41 that 

T^f G G{G,Qi,l^e"'+P''^l). 

We will also view Tjy as an operator with values in C{G, Qi, 1(8)6*^"^''' (8)1)® C 
(in the obvious way), and hence in 7^^. 

Lemma 5.1. For K {—{iv + pj)) > 0, the operator 

: C^{din) ^ 

intertwines vr' and tTj^. 

Proof. Let us set i'' = — (ii/ + p'j). Let h £ G. Then 

T^{h.f){g)= [ AT{h)f{h*x)A^,{g*x)dfii{x) 

JdUi 

= A-T(h)[ f{x)A^,{g*h-*x)d^,i{x) 
Jani 

= A-T{h) [ f{x)A^,{{h-^gyx)d^ii{x), 
Jani 

i.e., 

(11) T^{h.f){g) = A-T{h)TM){h~'9)- 

□ 

Since = f Ej=i('^+^+l-2j)'^j. we do not have 3?((-iA+p9) > when 
A G 0;*, and the integral (jlOp does not converge. This means that the integral 
has to be interpreted in a suitable sense using analytic continuation in the 
parameter u. For this we recall that when i/ G C', the Riesz distribution 
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on V can be defined as the analytic continuation of the fohowing 
integral 

K+'a{F) = r^(o {u + f / F{x)AUx)dfii{x), F e S{V), 

where S{V) is the Schwartz space of V, and that it has support in diO, 
(cf. [IshOOl Theorem 5.1 and 5.2], where the integral is actually defined 
over Oi = {x € 9/0 | A(;)(a;) / 0}, but fnidiQ \ Oi) = 0). The restriction 
(denoted by the same symbol) to the open set V>; is then a distribution 
with support in the submanifold We can therefore consider the vertical 
restrictions R^^^u I^jQ (cf. Appendix A). Since R^^j^m is a measure with 

support on for JRi^ > 0, these restrictions do not depend on the choice 
of a tubular neighbourhood (cf. Proposition IA.9P . For 3f?(— (ii/ + p^)) > 0, 
we have 

T^f{g) = r^,o {-{iv + p[) + f ) A-¥(5) {fig-*-)) . 

Hence, we can let the right hand side define an analytic continuation of the 
integrals T^f{g). It is defined on the complement Z of the set of poles of the 
meromorphic function Tq(i) ( — (iv + pj))- Since for fixed / G C^{diQ) the 
map g i-^ f{g~*-) is continuous, the function g i— > T,^{f){g) is continuous. 

Proposition 5.2. For any u £ Z, 

T^f G C{G,Qi,l^e'''+Pi^l), 

and the operator 

: c^idiQ) ^ n.. 

intertwines tt' and vr^. 

Proof. The equation describing the (5/-equivariance as well as eq. (fTT]l are 
analytic in the parameter f. Hence they hold by analytic continuation since 
they hold on the open set where 3f?(— (iiv + p^)) > 0. □ 

We now recall, in order to fix the notations, the definition of the spherical 
Fourier transform on ^}^^\ If / is a continuous function with compact support 
on O*^') which is ii'^'^-invariant, its spherical Fourier transform is 



fiu) = l^^^^f{x)^^^l^^,,ix)d^X 



where G (o^c)* and p^'^ = j Z^j=i (2i — I — ^)Sj- Since / has compact 
support, the function / is holomorphic on (a/c)*. For latter use we also 
recall the inversion for mula fo r / G C^{n^^^)^^'^ and A G (cf. [FK941 
Theorem XIV.5.3] and |Hel84[ Ch. Ill, Theorem 7.4]): 



(12) fix) =4^ / fiiX)^'lnAx) 



(0 / (^) dX 



where dX is the Lebesgue measure on ~ M', c^^\X) is Harish Chandra's 
c-function for Q^''\ and Cq^ is a positive constant. 

Now let / G C^{diVL)^, and observe that J^('), being a fibre of di9. Ui, 
is closed in and hence / 1^(0 (sometimes still denoted by /) has compact 



14 STEPHANE MERIGON AND HENRIK SEPPANEN 

support in Moreover, since any k € K^^^ extends to an element of K, 
the function / \q(i) is -invariant. 

Proposition 5.3. If f £ CQ°{diQ) is K-invariant then 

(13) T^fig) = - f )A_(,,+p,)(ff*e) 

Proof. Let us set v>' = —{iv + p'l) in the following. Again, by analytic 
continuation, it suffices to prove the equality for "^{v') > 0. Since / and in 
are X-invariant one has for all k m. 

Tuf{9)= f f{k-^x)A^,{g*x)dM^)= [ f{x)A^,{g*kx)dfii{x). 
JdQ.1 Jafli 

Hence 

T^f{g)= [ T^f{g)dk= I f{x)([ A^,{g*kx)dk] dfii{x). 

JK JdQi \Jk / 

Writing g* = th, t e NA, h e K,we have 

Ajjiithkx) = A^i {hkx) Aiji {te) = A^i {hkx) A^i {g* e) , 
and using the left invar iance of the Haar measure of K, 

/ A^,{g*kx)dk = / A^:{hkx)dkA^i{g* e) 
Jk Jk 

= / A^:{kx)dkA^:{g*e) 
Jk 

= $^.'(x)A^/(c/*e), 

hence, 

T^.f{g) = [ f{x)^^>{x)diii{x)A^,{g*e). 
Now Upmeier and Arazy's polar decomposition ^ for m yields 

Tr.f{g) = [ f{x)^^,{x)A%{x)d'i\A^.{g*e), 
and by Theorem l3.lt 

= l2K-u' + p^''^-^i)A^.{g*e). 
Since -p'i + p^^ - ^ = we eventually get 

T.f{g) = 7^U^;)/(^^ - f )A_(..+^;)(<7*e). 

□ 

In the following we set, 
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and we note that it defines a meromorphic function whose poles are those 
of rfi{—{iu + p'j) + y). The following lemma will be used in the proof of 
the Plancherel formula. 

Lemma 5.4 (Inversion formula). Let f € C^{di^l)^ and let A € o^*. Then 
for X € 

d\ 



^0 



(0 



-{i\+p[)i 



f{x) = c, 
Proof. We have 

(14) /a|(zA) = m - f ) = /(A)(7^U.;))"' 



cW(A)| 



hence the inversion formula (I12|) applied to the function /A^^^ gives the 



We now state the main result of the article. Recall the notations from 
the end of section [H 

Theorem 5.5 (The Plancherel Theorem). Let p be the measure on a^*/W; 
defined by 



dpiX) 



.(0 



:dX. 



Then there exists an isomorphism of unitary representations 

T:(7:\L\din)) (^J^jxdpW, J^^H^dpiX)^ , 
such that for every f e C^idiQ), {Tf)x = Txf . 

Proof. First we prove that for any ET-invariant function / in C^{diQ), 



(15) 



d,n 



\f{x)\^ dni{x) 



/(A) dp{\). 



For this purpose we use the polar decomposition for in and the inversion 
formula of Lemma 15. 4[ Then 



\f{x)\ diii{x) 



|/(x)|2A(^)(x)dl')x 



/(x)A(?)(x)c?)/ /(A)cl>- (x)(7-,^,p 



(0 



„(0 



dX 



:«(A) 



/(A)(75 



/(^)(7^Up;))"VA,M^A)^ 



^^J(x)A(})(x)cI>«^^^,,(x)dl')x 



C^dX 

c(0(A)f 



3(0 (A) I ^ 



/(A) 



c^dX 



(0 

')'-(iA+p{) 



,M<A) 



2 ■ 
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In the last equality we have used again the formula ([T] 

The next step is to prove that for a dense subset of functions / in Cg°((?;r2), 
the identity 

/ |/(x)pd^,(x)= / \T^f\ldp{X) 
JdiU J at 



holds. 

Recall that L^{G) is a Banach *-algebra when equipped with convolution 
as multiplication, and <f*{g) := tp{g~'^). Let L^(G)* denote the (commuta- 
tive) closed subalgebra of left and right K -invariant functions in L}{G)'^ . 
There is a natural projection L^{G) — > L^{G)'^, 



K JK 



(p{k-^ ^ ■ k2)dkidk2. 



For a unitary representation (r, J^) of G, there is a *-representation (also 
denoted by r) of L^{G) on ,3^ given by 



-{ip)v := / ip{g)T{g)vdg, 
Jg 



The representations of K and L^{G) are related by 

(16) T{ki)T{ip)T{k2)=T{ip{k^^ -k:^^)), ipeL\G), kiM^K. 



The subspace of i^-invariants is invariant under L^{G)'^ . From (fT6l 
it follows that for any tp G L^{G), and u,v ^ Jif^, 



(17) {t{ip)u,v) = {t{ip*)u,v). 

Let ^ be the ii'-invariant cyclic vector in L'^{diQ,). We claim that there 
exists a sequence {^n}^i C Cq°(9/0)^, such that ^ ^ in L'^{diQ). To 
see this, we can first choose a sequence {Cn}^=i ^ CQ°{diQ,) that converges 
to ^. Next, observe that the orthogonal projection P : Lp'{diVL) — > L'^{diCl) is 
given by / I— > f{k~^-)dk. Then P{f) is smooth if / is smooth. Moreover, 
supp / is contained in the image of the map K x supp / — > di^l, {k, x) — > kx. 
It follows that P{C^{din)) C G^{din)^. Hence, the claim holds with 

'■= -P(Cn)- The subspace 



^0-.= {vr'(/)e„ I /eCo~(G),nGN} 
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is then dense in L'^{di9). For (^s G C°°(G), n G N, we have, by ^ and p^ . 

J a* 




{irxi^* * ^)Tx{^n),Tx{U))xdp{X) 

Mv)Tx{Cn),A'P)Tx{^n))xdp{\) 



= [ {Tx{7T\v)Cn),Tx{7T'{^)^n))xdp{X). 
J a* 

Hence, the operator T defined on by T(7r'((/?)^„) = {'n'x{ip)T\{^n))x ex- 
tends uniquely to a G-equivariant isometric operator 

T:L\din)^ [ HxdpiX). 
J a* 

It now only remains to prove the surjectivity of T. Assume therefore 
that {tix)x is orthogonal to the image of T. Then for all in L^{G) and 
/iGLi(G)#, 

f {TTxiv * h){TOx,Vx)xdp{X) = 0, 

J at 

i.e., 

/ h{X){7rx{v){T0x,Vx)xdp{X) = 0, 

where h{X) is the Gelfand transform of h restricted to a^*. Recall that the 
set of bounded spherical functions can be identified with the character space 
of and hence the image of L^(G)* under the Gelfand transform 

separates points in this space. It thus follows from the Stone- Weierstrass 
Theorem that the functions h are dense in the space of continuous functions 
on that are invariant under the action of Wi.. Hence {T^xiDiTOx: ^a)a = 
p-almost everywhere. By separability of L^{G), there is a set U with 
p{al\U) = such that for all / in L\G) and XgU, {TTx{f)iTC)x,r]x)x = 0. 
By cyclicity of {TS,)x (note that {TS^)x is non-zero p-almost everywhere), ijx 
is zero p- almost everywhere. 

□ 

Remark 5.6. We want to point out that it is actually not necessary to prove 
the analytic continuation of (and hence to use the theory of Riesz distri- 
butions) to derive the decomposition of vr' (however, the natural operator T 
above is then replaced by an abstract one). Indeed, by the Cartan-Helgason 
theorem ( [Hel84^ Ch. Ill, Lemma 3.6]) we have Ti.^ = Cvx when A G a^, 
and hence we can set 

Tx-.C^idiQf ^n^, f^fiX)vx, 



18 STEPHANE MERIGON AND HENRIK SEPPANEN 

and by ([IS]) we thus obtain an operator T : L'^{diVt)^ H^dp{X). 

Assume that we can prove that T intertwines the actions of C^{G)'^. Then 
for if E C^{G)*, 

f{X)m)x\ldp{X), 

where ip is defined by Trx{f)vx = (p{\)vx. The proof of |Sep07b Theorem 
10] shows that the decomposition of vr' then foUows. We now prove the 
intertwining property. It is equivalent to the equahty 

(18) T^fw = nxrnx). 

Let u (£ CK Then for / G C^idi^) and ip G C^(G)# we have, where 
u' = -{if + p\), 

7r^(v?) (A^/((-)*e) (g) 1) (5) = / ip{h)A,,,{g*h'*e)<E)A'T(he)dh 

Jg 

ip{h)A-f{he) (^j A^,{g*kh-*e)dk^ ®ldh 



G 



ip{h)A-'-{he) 



j A^,{kh~*e)dk^ dh {A^'{g*e) ® 1) 



Lp{h)A'^{he)^^'{h~*e)dh (A^/(5*e) ® 1) , 

'G 
i.e., 

where <f{v) is holomorphic on C'. If (ii/' + p^)) > 0, then, by Lemma [5. 1^ 
the operator intertwines the actions of C^(G)*, and hence 

Thus p8|) follows by analytic continuation. 

Appendix A. Restrictions of distributions 

Let X be a smooth n-dimensional manifold. Let &{X) denote the space 
of compactly supported smooth functions on X, i.e., the test functions on 
X. For any chart (F, </>), compact subset K C V := (piV), and € N, 
consider the seminorm 

(19) py^K^^{f):= ^ sup,g^|Z)"(/o,^-i)(x)| 

QeN",|a|<Af 

on the space of smooth functions on X. Here Z?" := „ af a„ for a = 

(ai, . . . ,a„), and |a| = |ai| + • ■ ■ + For a compact set ZC C X, let 

Si{K) be the space of smooth functions with support in K equipped with 
the topology induced by the above seminorms. We recall that a distribution 
on A is a continuous functional on &{X) = UkS>{K) equipped with the 
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inductive limit topology. We let !^'{X) denote the space of distributions on 
X. Let {Ui} be an open cover of X. Then a linear functional on &{X) is 
continuous if and only if its restriction to every &(JJi) is continuous. 

We will now construct restrictions to a closed submanifold Y of distribu- 
tions that have support on Y. To have a well-defined notion of restriction, 
one can not permit arbitrary extensions to X of test functions on Y. Instead, 
we will require the extension to be locally constant along some predescribed 
direction. This can be made precise using tubular neighbourhoods. 

Definition A.l. Let X be a smooth n-dimensional manifold, and let Y be 
a fc-dimensional submanifold. A tubular neighbourhood of y in X consists of 
a smooth vector bundle tt : E ^ Y, an open neighbourhood Z of the image, 
(^EiY), of the zero section in E, and a diffeomorphism f : Z ^ O onto an 
open set O Q X containing Y, such that the diagram 



commutes. Here j : Y ^ X is the inclusion map. 

Remark A. 2. Any closed submanifold y of A admits a tubular neighbour- 
hood (cf. [Lan721 Ch. IV, F, Thm. 9]). Any sphtting of the tangent bundle 
of X over Y, T{X) |y= T{Y) © E, gives such a vector bundle E. In partic- 
ular, given a Riemannian metric on X, E can be chosen as the orthogonal 
complement to T{Y) in T{X) |y. 

Since the concepts we are dealing with are of a local nature we can without 
loss of generality assume that X = Z itself is a tubular neighbourhood of 
Y. 

Definition A. 3. A function / on A is said to be locally vertically constant 
around Y, l.v.c, if for any x £ Y, there exists an open neighbourhood Wx 
of X in A such that for y G Wx, f{y) = /(7r(y)). Moreover, if g is a function 
on Y, and / |y= g, f is called an l.v.c. extension of g. 

Lemma A. 4. 

(i) Any test function iponY admits a l.v.c extension (p £ &{X). 
(a) An l.v.c. function f that vanishes on Y vanishes on some neighbour- 
hood of Y . 

Proof. Since supp(/9 is compact, it can be covered by finitely many open 
neighbourhoods Oi, . . . , On diffeomorphic to products UixVi <Z M'^ffiM"'"*^, 
where Ui is open in M.^ and Vi is an open neighbourhood of in M"~'^, in such 
a way that vr corresponds to the projection onto the first coordinate. Let 
{il^i) be a smooth partition of unity on U = U^i7r{Oi) subordinate to the 
cover 7r(Oj), i = 1, . . . , N, and let ifi = ip^i. Then each ipi can be identified 
with a test function on Ui, and by multiplying this with a test function on 
Vi which is 1 on some neighbourhood of 0, we obtain a l.v.c. extension ipi 
of ipi. Then ^ = ^ (^j is a l.v.c extension of if. This proves (i). For (ii) just 
observe that if / is an l.v.c function that vanishes on Y, then Y is in the 
complement of the support of /. □ 
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Assume now that u S Si'{X) has support on the submanifold Y . Then l.v.c. 
test functions that vanish on a neighbourhood of Y are in the kernel of ii, 
and the preceding lemma enables us to make the following definition. 

Definition A. 5. Let u € &'{X) be a distribution with support on Y . The 
vertical restriction, u |y, of u to y is the unique distribution on Y that 
satisfies 

^ \y \y) = u{ip), 
for any (p € &{X) which is l.v.c. around Y. 

To see that the functional u \y really is a distribution on Y, it suffices 
to verify the continuity for test functions with support in trivialising open 
sets. In this case the verification is straightforward using the l.v.c extension 
from the proof of Lemma lA.4i 

Remark A. 6. Note that the vertical restriction depends on the choice of 
tubular neighbourhood, i.e., on a choice of complement E in the splitting 
of vector bundles in Remark IA.2i However, when u is a measure on X 
with support on Y, the vertical restriction u |y is u itself, now viewed as a 
distribution on Y. 

We now consider holomorphic families of distributions and their properties 
under restriction. 

Definition A. 7. Let C C™ be an open set, and let {u^jzgQ be a family 
of distributions on the smooth manifold X. Then this family is called a 
holomorphic family of distributions if the map z u^{^) is holomorphic on 
O for every ip £ &{X). 

Remark A. 8. It follows immediately from Definition IA.5I that if {u^j^gn 
is a holomorphic family of distributions with support on Y, then the family 
{u^ lyjzgn is a holomorphic family of distributions on Y. 

Proposition A. 9. Let 0, C C" be open and connected, and let {u^}z£n 
a holomorphic family of distributions on X with support on Y. Assume that 
there exists an open subset f/ C fi, such that is a measure with support 
on Y for z E U. Then the whole family {u^ lyjzgn is independent of E. 
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